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© Automatic welding. 

© Apparatus to control the automatic placing of material 
along a Juncfton between surfaces with reference to the 

movable to deposit material progressively along the junction 
movemX'of th ' ^ ~ ""^' to he 
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form and posit.on. means responsive to said image portion 
to der ve sa .d control signal to control the means?! Cos." 
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AUTOMATIC WELDING 
This invention relates to the automatic control of the placing 
of material at a junction between surfaces and more particularly 
to the placing of a weld to join the surfaces* 

The material may be a weld to be formed at the junction of 

05 the surface. The material may be an adhesive or sealant to be 

placed at the junction of the surfaces. The automatic control is 
to respond to the position and form of the junction to the material 
correctly. One use of the invention is the automatic arc welding 
together of sheet metal pressings. 

10 Automatic control of the placing of material such as a weld 

bead with reference to the form and position of a junction between 
surfaces is a difficult procedure when the surfaces are other than 
perfectly regular and predictable. A skilled human welder makes 
many rapid changes in torch position, speed etc. while making a 

15 weld to take account of variations in the separation between 

surfaces at the junction, distortions in the surfaces etc. The 
human welder bases these changes on many pieces of information 
which are assessed rapidly enough to be used to control the torch 
as it moves along the weld. An automatic process to reproduce 

20 these rapid changes must have a very large information processing 
capacity to imitate the human. To operate such a large processing 
capacity quickly enough to make appropriate real-time changes in 
weld conditions is expensive and has hitherto prevented the intro- 
duction of automatic welding techniques that are fully competitive 

25 with the human welder. 

In a paper published in March 1982 various methods of arc- 
welditag metal pressings were assessed. (Clocks in, Barratt, Davey, 
Morgan and: Vidler, Visually guided arc-welding of thin sheet steel 
pressings. Proc, Int. Symp. Industrial Robots, 12, Paris). The 

30 desirability of achieving a weld in one pass over each production 
item is clear but raises many problems of control because of seam 
variation and the harsh welding environment. A general form for 
an apparatus to achieve such one-pass welding, following a teaching 
passover a master workpiece, is set out in the above paper and a 
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form of sensor to measure the three-dimensional geometry of the 
workpiece surface is outlined as having an infra-red GaAlAs laser 
with a cylindrical lens to cast a sheet of light on a workpiece. 
A solid state camera views the stripe of light on the workpiece at 
the intersection with the light sheet. A filter on the camera 
reduces the effect of visible arc emission. The camera is not 
severely affected by electromagnetic emission even when 3 cm from 
the arc and a metal shield protects the arrangement from molten 
metal. The camera shield is directed to a cleaning station when 
it is necessary to remove weld spatter. However the apparatus, is 
too slow for effective use, although reasonably accurate, when 
faced with a production workpiece which is significantly different 
from the master workpiece. 

It is an object of the invention to provide an apparatus and 
a method to control the automatic placing of material at a junction 
between surfaces with regard to the position and form of the 
junction which has a faster response. 

In particular automatic arc welding is envisaged. 

According to the invention there is provided apparatus to 
control the automatic placing of material along a junction between 
surfaces with reference .to. the form and position of the junction 
including means controllably movable to deposit material progres- 
sively along the junction' in response to a. control signal, means 
linked to the movement of the means to deposit material to produce 
an image of the. surfaces and junction, means to extract from the 
image that portion unambiguously defining the junction form and 
position, means responsive to said image portion to derive said 
control signal to control the means to deposit material to move in 
a required manner along said junction to deposit material. 

The image may be an array of pixel elements identified by a 
set of row indices and a set of column indices and the extracted 
portion may be an array of pixels, one pixel for each of one of 
said sets of indices. The image may be further simplified t at 
least one straight— line segment defined by endpoints in the array. 

The image may be processed by a filter to remove from the 
image inf rmation produced by illumination at other than a 
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specific wavelength, which filter includes a tapped delay to 
provide a specific impulse response. 

The apparatus may include a light source and a camera with 
the light source arranged to produce a shield of light inclined to 
the axis of the camera together with a welding torch having an 
axis which is parallel to that of the camera. 

According to the invention there is also provided a method of 
controlling an automatic arc welding process at a junction of 
surfaces including producing an image of a welding area containing 
the junction and illuminated by a light sheet, extracting from the 
image that portion representing the form and position of the 
junction and deriving from said image portion control information 
ta». control the welding, process to operate along said junction. 

Embodiments of the invention will now be described with 
reference to the accompanying drawings in which:- 

Figure 1 shows a block schematic of a system embodying the 
invention. 

Figure 2 shows a sensor in outline. 

Figure 3 shows response curves, for a semiconductor laser at 
Figure 3a, a filter at Figure 3b and a camera at Figure 3c. 

Figure 4 shows at Figure 4a, 4b, 4c and 4d images produced by 
the camera of the sensor of Figure 2 when viewing various joints 
illuminated by the light sheet of Figure 2. 

Figure 5 shows the response of a filter. 

Figure 6 shows stages in the processing of image information. 
Figure 7 shows how "segments" are fitted to a processed image 
to represent a joint for identification. 

Figure 8 is a table showing how lap joints are identified and 
classified. 



Figure 9 shows how a master and a production assembly are 
compared. 

Figure 10 is an axial cross-section of an assembly of a 
welding torch with cameras and light sources. 

Figure 11 is a view of parts of the assembly shown in 
35 Figure 10. ' 
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Figure 12 is a further block schematic circuit of a control 
arrangement embodying the invention. 

Figures 13 and 14 are representations of a welding area and 
picked up illumination information, helpful in understanding the 
action of the invention. 

Figure 15 shows a response of a filter used in an embodiment 
of the invention and, 

Figure 16 shows the action of such a filter. 

Figure 1 shows a block schematic of a system embodying the 
Invention. The system consists of a conventional robot R equipped 
for automatic KEG welding with an assembly of a torch WT, a control 
sub-unit CS and a vision arrangement including a combined camera 
and illumination projector CP mounted on torch WT and vision 
sub— unit VS, The assembly is supplied with power, welding wire, 
weld arc shielding gas and cooling fluid through a multi-way 
duct UC. Signal connections also extend to the assembly 
conveniently along multi-way duct UC. The camera and projector 
are attached to the welding torch in a set relationship. The 
robot is an ASEA (RTM) IRb-6 industrial robot with 5 degrees of 
freedom and a payload of 6Kg at rest. The IRb-6 robot is equipped 
with an Aga (RTM)_ CW9 MLG— welding set, and both robot and set are 
controlled by the standard ASEA controller containing an X8008 
microprocessor. Minor readily produced Interface modifications to 
the welding set and 8008 program are required. 

The control sub-unit need only be briefly described. It 
consists of an LSI-11/23 which is connected to the ASEA controller 
by a 9600 baud serial line. As the ASEA controller is not equipped 
with a co-ordinate conversion capability a kinematic model is 
included in the control sub-unit to permit programming and execu- 
ting ASEA programs in a number of different cartesian "world" 
co-ordinate systems. This implementation of co-ordinate conver- 
sion w rks in real— time, does not exhibit cumulative errors over 
successive forward and Inverse conversions, deals with arbitrary 
offsets f multiple to Is about the wrist, is easily calibrated by 
the user, and permits arbitrary geometrical transformations and 
f llowing of arbitrary space curve paths, interpolating in world 
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co-ordinates. Because the vision eub-unit is considered as a tool 
on the robot, the control sub-unit also deals with arbitrary 
transformations between image co-ordinates, spatial co-ordinates, 
and corresponding robot joint co-ordinates. 

The vision sub-unit VS consists of PDP-11/23 computer 
equipped with a sensor interface. The sensor indicated generally 
at CP in Figure 1 and consisting of a GaAlAs infra-red laser 
diode LD emitting at 830 nm and a rectangular CCD camera with a 
narrowband optical filter CF, is shown in outline in Figure 2. 
The laser diode emits through a narrow slit and cylindrical lens 
to cast a sheet of light SL on the workpiece UP. The intersection 
of the light sheet SL and the surface of workpiece VP, when viewed 
from a different regard with the camera, forms a curved or 
straight-line stripe LS that depicts the layout of a particular 
cross-section of the surface relief of the workpiece. The camera 
and diode and lens are arranged at an angle to one another, T in 
Figure 2, so that their axes cross in the general region of the 
surface of workpiece WP and near to where the torch produces the 
weld. The camera generates a PAL standard video signal which is 
digitised and stored as a 256 x 256 array of 8-bit pixels ±a known 
manner. A line 10 pixels in length subtends approximately 1 mm on 
an inspected surface WP on which the laser is focussed. The 
sensor is fixed to the end-effector of the robot, near the welding 
torch. The. output of the CCD chip is little affected by spurious 
pick-up of the electromagnetic radiation from the welding arc. 
The camera is fitted with a narrowband optical filter having a 
spectral halfwidth of 10 nm centred around 830 nm which effec- 
tively removes much of the visible arc emission from the image. A 
metal shield (not shown) protects the sensor assembly from molten 
mental splatter and also acts as a heat sink. The response curves 
of the laser LD, filter and camera are shown in Figures 3a, 3b 
and 3c respectively. 

The sensor operates on the principle of active ^angulation ' 
ranging, a technique which many researchers have used for obtain- 
ing information about the three dimensional layout of surfaces in 
a scene (for example, Agin, C.J. and Binford, T.0. (1973), Computer 
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description of curved objects Proc. Int. Conf • Artificial Intelli- 
gence 3, Stanford. Popplestone, R.J. and Ambler, A. P. (1977). 
Forming body models from range data, Research Report 46, Department 
of Artificial Intelligence, University of Edinburgh. Hill, J. and 

05 Park, W.T. (1979). Real-time control of a robot with a mobile 
camera, Proc. Int. Symp. Industrial Robots 9, Washington D.C.) 

The camera images shown in Figure A result when the light 
sheet SL crosses different types of weld joints and Is viewed 
along arrow X. The closer to the top of the Image that a point on 

10 the stripe lies, the further away is the surface to which the 

light is projected. Gaps such as G are also shown in the images • 
The purpose of the sensor described here is not to examine the 
molten weld pool, but to provide a means of determining the 
geometrical relationship betweeen the torch and the seam. The 
15 actual image processing operations used are discussed below. 

This apparatus is usable to enhance the conventional robot 
teaching/welding sequence by performing the following three phases 
not always as specific separate actions:— 
Phase 1 

20 Manual teaching of weld seams on a pre-production master 

assembly. Straight-line seams are more easily taught by speci- 
fying a pair of endpoints. The kinematics model automatically 
derives the straight line in space connecting the endpoints. It 
; Is simple to extend the sub-unit to Interpolate circular paths 
25 from three points and curved spline paths from four or more 

points. More importantly, perfectly taught trajectories will not 
be necessary when the sensory sub-unit is able to adjust the 
trajectory to fit the workpiece at hand. Furthermore, weaves can 
be taught by specifying a weave amplitude, the kinematic model 
30 then automatically derives a sinusoidal weaving path lying in the 
-, - . , plane normal to the torch. 
. Phase 2 

Aut matic visual survey of taught points on the master 
assembly. The control system runs the taught robot program 
' .v. 35 unattended on the pre-production master assembly, and uses the 

sensor sub-unit t identify several items of reference information 
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for each taught point, including: the type of weld joint at each 
point, the position of the torch relative to terrain* near the 
seam, and a metal thickness factor for lap seams. Gap width is 
known to be negligible on master assemblies. The system also 
generates its own idea of where the torch ought to be positioned, 

but this capability is not used in the present embodiment. This 
entire step takes about 2 seconds per taught point. For a single 
straight-line seam, this amounts to at most 5 seconds to survey 
the seam. This inspection phase can be merged with the manual 
teaching phase. 
Phase 3 

Welding with visual correction of taught points and welding 
parameters. This phase is repeated for each production assembly, 
and can involve making two passes over the workpiece: inspec- 
tion (3a) and welding (3b). For phase 3a, the control sub-unit 
drives the robot to the taught seam endpoints. At each endpoint, 
the sensor sub-unit provides information to measure the width of 
the seam gap, and the position of the torch relative to the seam. 
This information is compared with the reference information derived 
previously from the pre-production master assembly. Discrepancies 
between the master and the production assembly are treated as 
error signals.. The error signals are fed back to the control 
sub-unit, which automatically derives a new corrected program to 

weld the seam (3b),. To save time when two passes are needed, the 
Inspection pasfe follows the taught program along the seam, and the 
welding pass follows the corrected program in the reverse direction. 

The following error ^signals are detected and corrected: standoff 
error (torch altitude above the seam along the torch axis), lateral 
error (deviation across seam from centre of seam), and gap width GW. 

The gap width and given metal thickness are used to calculate the 
wire feed speed and torch travel speed using a parametric welding 
model based on Hunter, J.J. Bryce, G.W. and Doherty, J. (1980) 
On-line control control of the arc welding process, Proc. Int. 
Conf. Developments; in Mechanised, Automated, and Robotic Welding, 
London (Welding Institute). In one embodiment the time taken for 
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each seam ±8 about 2 seconds per po±nt analysed plus the time 
taken for the actual welding. 

Several points about the above procedure should be noted. 

First, information about metal thickness must be supplied to 
the system by the operator. This is a reasonable condition, as 
metal thickness is specified when the assembly is designed, and it 
is known to the operator. Visual detection of a metal thickness 
factor during Phase 2 above is used for the purpose of detecting 
gaps in lap joints, as described below. 

Second, path correction and welding, Phase 3 above, forces 
the system in a simple form to make two separate passes over the 
seam: and inspection pass (3a) and a welding pass (3b). The 
consequence is that variations due to thermal distortion during 
welding are ignored . This has proved not to "be a problem in 
trials, but simultaneous inspection and welding would be required 
to increase the yield of correctly welded seams in a production 
environment. The reason for correction and welding in separate 
passes is that the image processing alogorithms in the simple form 
are implemented wholly in software, and at 2 seconds to analyse in 
image, this is too slow for adequate simultaneous closed-loop 
control* For a maximum travel speed of 20 mm/s, and correcting 
every 2 mm along the seam, an image processing time of less 
than 100 ms is required. A more specialised image processor that 
enables the system to attain this cycle time is described below. 
Finally, the image processing algorithm^ .have . been tested in the 
presence of arc emissions with little ill effect, but exhaustive 
tests may be needed to determine whether the algorithms need to be 
more robust. 

The software for the sensor sub-unit is written mostly in 
Pascal (Jensen, K and Wirth, N (1974) PASCAL User Manual and 
Report, Lecture Notes in Computer Science, 18, (Sprtnger-Verlag) ) • 
The Pascal compiler generates PDP-11 machine instructions, but 
about 10 percent of the system is hand coded in PDP-11 instructions 
to decrease the time consumed for image processing. Three levels 
of representation aire used by the sensor algorithms. The lowest 
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level is the Image itself, which is represented as a 256 x 256 
array I of the integers in the range 0 to 255 called pixels. 
Pixels are addressed by a row index and a column index, and the 
pixel at row r and column c is denoted I[r,c] . The next level of 
representation is the "hit list", which is a 1 x 256 array H of 
row indices in the range 0..255. The ith element of H is 
denoted H[i] . Given an image I, containing a stripe as described 
in the previous section, H[iJ is defined to be the row index where 
the stripe in the image intersects the ith column of I. If no 
part of the stripe intersects column i, then H[i] is undefined 
(assigned a unique value) . Array H can be considered as a partial 
function of i. that represents the stripe, as the stripe can 
intersect a given column at most once. 

The final level of representation is an array S of segments, 
where each segment represents a straight-line segment lying in I. 
Components of a segment are the row and column indices of the 
endpoints, the length, and the slope. A stripe in the image is 
represented as an array of segments. Given endpoints (rl,cl) 
and (r2,c2), the slope is defined as (r2-rl)/(c2-cl) . As row 
indices increase going down the image, the sign of the slope will 
be inverted from the normal interpretation. The slope of the 
stripe is an indication of the slant of a surface in depth along 
one axis . 

If two points (rl,cl) and (r2,c2) lie on the image of the 
stripe, then the distance between the points on the workpiece 
surface to which these image points project can be calculated. 
Recall that for the sensor configuration used, displacements of 
the workpiece along the optical axis cause vertical displacements 
in the image of the stripe. Let Sr, Sc, and Sz be scaling factors 
having units of mm/pixelwidth. Sr relates differences in row 
indices to vertical displacements in space, normal to the optical 
axis. Sc relates differences in column indices to horizontal 
displacements in space, normal to the optical axis. Sz relates 
differences in row indices to displacements in space along the 
optical axis. These scaling factors are automatically calculated 
when the system is calibrated, and they are assumed constant 
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everywhere within a certain working area of the image. The 
distance on the workpiece (in mm) between (rl,cl) and (r2,c2) both 
lying on a reflection of the laser light is approximated by:- 

(Sr*(r2-rl)**2 + Sc*(c2-cl)**2 + Sz*(r2-rl)**2) ** (0.5). 

The sensor software provides functions for capturing images 
of stripes, transf otrming one representation into another, storing 
and comparing segments, and computing the error signals that are 
fed back to the control computer. Consider first the image 
processing operations carried out for teaching seam endpoints 
(Phase 1 above). First, the robot drives the camera so that the 
endpoint of the seam is in the centre of the image. This is done 
by commanding the control sub-unit to drive the robot so that the 
camera optical axis is co-linear with the axis of where the welding 
torch would be if the tip of the torch was positioned at the 
endpoint, normal to the seam. Provided that the fixed relation- 
ship between the welding torch axis and the optical axis is known, 
then the kinematic model is responsible for carrying out this 
transformation and robot movement. Next, an image is captured 
into array I, and the sensor sub— unit must now carry out some 
computations and store in a database some information associated 
with the given endpoint. The computations are divided into five 
steps summarised as follows: 

1. Construct the "hit list". This reduces the amount of informa- 
tion that encodes the stripe from the 65536 integers in the 
image to the 256 integers in the hit list. 

2. 'Clean' the hit list to remove noise and hypothesise missing 
data. 

3. Fit line segments to the hit list. This reduces the informa- 
tion still further to about one to ten segments. 

4. Identify the type of joint from the configuration of line 
segments. This locates the seam and represents the joint as 
a single label. The joint label is associated with the 

... endpoint. 

5. Determine the vector offset from the torch to the seam. This 
computation is actually performed by the control sub-unit, 



01 08599 



- li - 



since it involves transformations from one co-ordinate system 
to another. 

As Phase 1 (above) is being considered, the vector offset is 
a relationship determined during teaching, so it will be associated 
with the endpoint as the 'ideal' offset to be repeated for the 
production assemblies. 

For inspecting production assemblies prior to welding 
(Phase 3a above) the same steps as above are followed. However, 
once the vector offset is computed, it is compared with the 'ideal' 
offset in the database previously computed when the seam was 
taught on the pre-production master assembly (Phase 1). Any 
discrepancy is computed and the difference is fed back to the 
control sub-unit as a. .positional error. The error is used by the 
control sub-unit to construct the corrected welding program. The 
image processing steps are now described in detail. 
Constructing the Hit List 

To obtain element H[i] , the pixels in column 1 of the image 
are convolved with a discrete linear filter mask/ Let the mask 
array M be ah array, indexed from ^ to w, of numbers called 
weights. The convolution response C[j] of row ± of image column 1. 
is defined as 

w 

C[j] - sum M[k] I[j+k,i] * ... • 

k=— w 

The row index where the maximum convolution response occurs over 
column i is assigned to H[i]. This process is repeated for each 
column in the image. 

Several criteria govern the choice of filter mask. The 
primary criterion is that the filter should give a maximum 
response to the cross-section of a stripe, thus discriminating the 
stripe from other sources of brightness such as noise, mutual 
illumination of one steel sheet by reflection of laser light from 
another, and any molten metal splatter shooting though the field 
of view having spectral components in the range 825 to 835 nm. 
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Another requirement was that a wholly software implementation of 
the hit detector should process an image in not more than a few 
seconds, and should imply a simple hardware design that would run 
at video frame rates. In examining typical images, it is found 

05 that brightnesses of non— stripe regions in the image could be as 
bright as the stripe. Furthermore, the absolute brightness along 
the stripe could fluctuate across the entire brightness range of 
the image. This ruled out simple thresholding and the use of 
histograms to select thresholds. It has been found that it is 

10 reasonable to assume that the width of the cross-section of the 
stripe (sampled along an image column) was defined to within 
fairly narrow bounds, and that the width did not change signifi- 
cantly when the slope of the stripe was non— zero. Thus we were 
led to using differencing techniques, and could assume a fixed 

15 maskwidth. Four techniques were implemented and tested, of which 
one was chosen for the present system. Each column of the image 
is convolved with a discrete approximation to the DOG (difference 
of gaussian) filter (Marr, D, and Hildreth, E (1980) Theory of 
edge detection, Proceedings of the Royal Society (B) , 207, 187-217). 

20 This particular discrete approximation, which is equivalent to a 
second differences filter of zero net gain* consists of a central 
positive region (width 3, gain 2) surrounded by a negative region 
(two, eachtwidth 3, gain -1). The DOG and its family of discrete 
approximations is sensitive to impulses having widths less than 

25 the width of the central region of the mask. Although the spatial 
frequency response of the DOG filter is well localised, large 
amplitude impulses have significant power not only in the high 
spatial frequencies but also in the portion of the spatial 
frequency spectrum where the DOG filter is normally tuned. In 

30 spite of this limitation, which has not yet caused problems, the 
implementation of the sensor sub-unit for this embodiment uses 
this filter. Performance of the filter in rejecting nearby n ise 
pulses (due to mutual illumination) is shown in Figure 5. 

Figures 5a and 5c show in outline the response of the filter 

35 for two different patterns of individual image column brightness, 
shown in Figures 5b and 5d_. In each case the horizontal axis is 
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image row index while the vertical axes are arbitrary units. 
Figure 5b shows a typical image column with a stripe image, the 
spike, against a moderate noise background. The filter response 
with such an image is shown in Figure 5a. When the image is 
contaminated with noise a typical image brightness pattern is as 
shown in Figure 5d_. Arrow P indicates the stripe spike, the 
adjacent spikes are noise of similar frequency to that of the 
laser light source. Figure 5£ shows that the filter, although 

affected by the noise still maximises the response to the stripe. 
Given this mask, convolution requires no multiplications, only 
nine additions and one binary shift. The system also improves 
performance by convolving only a certain sub-range of each image 
column, using a statistic based on previous hit values to define 
the centre and length of the sub-range for a given image column. 
Cleaning the Hits 

It is possible that hits can be incorrectly assigned by the 
hit detection filter, for example, when a non-stripe pulse in a 
given column appears more "stripe-like" in the image than the 
stripe pulse itself as shown in Fiugre 6a. The causes of such 
noise were mentioned above, and it is necessary to process the hit 
array to limit the effects of such noise. First the hit array is 
cleansed by removing noise points and then connecting small gaps 
if they seem to lie on the stripe. Distinguishing between pixels 
of hits correctly positioned on the stripe and those misplaced 
(Figure 6b) is made easier because the hit array encodes depth 
information, and tolerances on possible surface profiles are 
known. For example, any H[i] that differs from one of its 
neighbours by more than a threshold is considered "noise", because 
the smoothness (continuity) of the metal surface is the limiting 
factor. Any such noise elements are replaced by an "undefined" 
marker. Furthermore, any connected run of hits of less than 10 
pixels (1 mm on the metal surface) is marked as undefined, as it 
is impossible for any physical feature on the metal surface to 
give rise to such a pattern of hits. The result of these two 
operations is to leave a pattern of connected hits and connected 
runs of undefined elements (gaps). 
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The next step examines each gap to account for the possiblity 
of dropouts along the stripe. As shown in Figure 6d, the pixel 
brightness values of the two defined endpoints (K; K+8)of the gap 
are examined, and a threshold is defined as 70% of the minimum 
brightness of the two endpoints. Then, the brightness of each 
pixel on the straight line between the endpoints is compared with 
the threshold (Figure 6e) , and is re-classified as a hit if the 
pixel brightness exceeds the threshold (Figure 6c). This opera- 
tion serves to hypothesise hits within gaps in the cleaned hit 
list, and it is the last operation in the processing sequence to 
.examine the image array. In practice it is found that the number 
of pixels examined in this way varies from none to about a dozen. 

Fitting Segments 

Next, the system fits line segments to the remaining 
connected runs of hits using a recursive descent algorithm 
(reported by Popplestone and Ambler (1977) supra) . A line is 
constructed between the two endpoints of a connected run of hits. 
The position on the run having the maximum deviation from the run 
to the straight line is noted, and if the deviation exceeds a 
threshold, then the run is split at this point (denoting a new 
endpoint) , and the algorithm is recursively applied to the run on 
each side of the split. The recursion terminates if either the 
maximum deviation does not exceed the threshold (indicating the 
presence of a line segment), or if the remaining run length is 
less than some threshold. It is realistic to employ such thres- 
holds because they are derived from known dimensional tolerances 
of the task. The result is an array containing one or more segment 
structures. The endpoints, slope and length of each segment are 
components of the structure. 

Identifying Joints 

Given the array of segments, the next step is to identify the 
type of joint depicted by the segments. There are three possible 
joints: butts, laps and T-joints. Gaps may be present, and the 
angle between the metal sheets making up the joint may vary. The 
array of segments is applied to a decision procedure that makes 
explicit some geometric and linguistic criteria that any legal 
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joint mast observe. Given a list of segments, the system first 
determines whether it represents a legal lap joint. If it fails 
the lap joint test, then the system determines whether it is a 
T-joint. If that test fails, then the system determines whether 
it is a butt joint. If that test fails, then the system cannot 
classify the joint, so another image is captured and the procedure 
repeats. If four successive images cannot be classified, then the 
system reports an error. The need to perform the tests in this 
order result from two important linguistic constraints: (a) lap 
joints contain a small fillet-like shape, so the presence of a lap 
joint somewhere in the list of segments rules out the possibility 
that a T-joint occurs anywhere in the list; (b) the butt joint is 
a special case of the lap joint, so the presence of a lap or 
T-joint rules out the possibility that a butt joint appears. 

The rules for the lap joint (the most complicated one) are 
given below. The rules draw on the properties of the segments 
and on some pre-defined constants calculated when the system is 
calibrated. Given a segment A, its properties are denoted as 
follows: slope (in the image plane) is denoted as A. slope (recall 
the 'upside-down' definition of array S above), length as A.length 
left endpoint as (A.rl.A.cl) and right endpoint as (A.r2,A.c2) 
The pair (TR,TC) is the row and column address of the pixel that 
corresponds to the location of the tip of the welding torch. As 
stated above, the camera and torch are separate tools having axes 
that do not necessarily intersect, so although the torch does not 
appear in the image, (TR.TC) represents where the torch would be 
on the workpiece if it were in position. Other constants are 
thresholds which represent allowable dimensional variation in the 
workpiece and tolerances of the task. The thresholds are TL (a 
length of 2 mm on the metal surface), TA (an angular measure of 15 
degrees in the image plane) , TG (a distance of 3 mm denoting the 
maximum tolerated gap width), and TT (a distance f 5 m denoting 
the maximum t lerated deviatibtf f rom the taught path). The three- 
dimensional distance in space denoted by a distance between two 
image points (rl.cl) and (r2,c2) that lie on the image of the 
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stripe is denoted D((rl,cl) , (r2,c2)) , and is computed in the same 
way as previously described. 

Figure 7 depicts how the light stripe reflects from a lap 
joint to form segments, and shows how the centre of the seam can 
be detected. Two representative segments, A and B, are sufficient 
to represent the lap joint. The right endpoint of segment A we 
call PI, and the left endpoint of segment B we call P2. We 
describe below how the system chooses the representative 
segments A and B given a list of segments. Segment E in 
Figure 7(a) results from the edge of the top surface. For the 
pre-production prototype without any gap, G in Figure 4, segment E 
should joint points PI and P2 as shown. However, the cut edges of 
the metal sheets are likely to be specular reflectors less than 2 mm 
thick, and therefore may produce a large bloom in the image where 
segment E should appear. To get around this problem, the system 
first chooses A and B as described below, and uses the distance 
between PI and P2 to represent E, rather than any image information 
that might appear. The hit hypothesiser will fill to every pixel 
on the straight line between PI and P2 only if no gap is present, 
which turns out to be the- proper thing to do, but the information 
is not trusted in this case. When inspecting the pre-production 
assembly, the length of E is proportional to the metal thickness, 
and angle 3 is used to describe the line that will need to be 
hypothesised when the production assembly (with possible gap) is 
inspected. Figure 7(b) shows how angle 3 is used for a lap with a 
gap* 

The circle in each set of segments denote the estimate, made 
in the system, of where the centre of the seam is. In Figure 7(a), 
the centre is the corner between the bottom sheet and the edge of 
the top sheet. This is point P2, the intersection of segment E 
and B. The same applies to Figure 7(b), but notice the portion of 
segments to the left of point P2«. This part of the segment is 
where the stripe is seen though the gap on the bottom surface. 

The lap joint can be approached from two possible directions, 
and a gap may appear, giving four possible classes of joint shape. 
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A different procedure Is u 8 ed to find the seam centre of each type 
of Joint shape. Figure 8 summarises the procedures as a table A 
list of segments derived from the image is defined to represent a 
lap joint if and only if two segments A and B of maximal length 
exist that satisfy the following rules: 

1. A minimum length condition: (A. length > TL) and 

(B.length > TL). This is to ensure that informative segments 
are used. 

2. Either (A.slope < 0) and (B.rl > A.r2), or (A.slope > 0) 
and (B.rl < A.r2). Recall that rl and r2 are row indicles, 
which decrease in value towards the top of the image. If 
this test succeeds, then it is established that we are 
approaching a lap joint from a legal direction. If the left 
side of the disjunction succeeds, then the A-aide of the 
joint is the top surface. If the right side succeeds, then 
the B-side is the top. 

3. The absolute value of the angle between A and B is lees 

than TA. This rule distinguishes laps from T-joints. 
A. The distance from (A.r2) to (B.rl) must not exceed TG but 
must be at least as great as the metal thickness. Note that 
this distance is an image row distance, which corresponds to 
a distance in depth along the optical axis. 
Other sets of rules are used to recognise T-joints and butt joints 
Once a list of segments is identified with a particular joint 

> type, the following ^formation is associated with the appropriate 
endpoint in the taught welding program: the slopes of A and B 
and an estimate of where the centre of seam is. A metal thickness 
factor (the distance between (A.r2) and (B.rl)) and angle a are 
also stored for lap joints. 

> Determining Offsets 

During the correction phase, information collected during the 
teach phase (slopes of legal A and B segments, estimate of seam 
centre, etc.) is considered as a model of what to look for when 
inspecting the production assembly J The system now attempts to 
match the model derived from the master assembly with the segments 
derived from an image of a joint on the product! n assembly. 



D 
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Figure 9 shows segments for a master assembly MA. and a production 
assembly PA. Let the following terms denote the components of the 
model: left slope (taught slope of A), right slope (taught slope 
of B), (cr,cc) (row and column indices of seam centre), and angle S 
(observed angle between bottom sheet and edge of laps) . Given a 
list of segments derived from the image of a joint on the produc- 
tion assembly, the system now attempts to find segments A and B 
that satisfy the following rules: 

1. The absolute angle between A.slope and left, slope does not 
exceed TA, and the absolute angle between B. slope and 
right. slope does not exceed TA. 

2. Compute an estimate (sr,sc) of the seam centre. Angle S is 
used to construct a seam centre for lap joints as previously 
described. Compare (sr,sc) with (cr.cc) to compute gap, 
standoff error, and lateral error. Figure 9 shows how this 
is done for the lap joint, and similar constructions hold for 
the T- joint and butt joint. It must be remembered that 
vertical displacements in segment representation denote 
displacements in deptch. Dimensions s_ and jt are the errors 
respectively along the torch/camera axis and laterally in the 
plane normal to the torch axis. Dimension h is the metal 
thickness factor, the projection of the distance between the 
top and bottom surfaces of the master assembly into the plane 
normal to the torch axis. Dimension £ is the equivalent of h 
but for the production assembly. If the actual metal thick- 
ness M is known then the gap for the production assembly is 
approximately M(g-h)/h. Gap width must not exceed TG, and 
standoff and lateral error must not exceed 5 mm. Note that 
gap width is defined along the horizontal image axis, 

3 indicating the projection of the gap onto a plane normal to 

the torch. This is an arbitrary choice. An alternative 
would be , to define gap as the distance from the edge of the 
top surface to the centre of the seam, for example. 
If segments cannot be found that match the model with the joint on 

l5 the pr duct ion assembly, then the system tries again with another 
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provided for the supply of compressed air but is not shown. Air 
thus supplied circulates through the peripheral part of the cylin- 
drical body and is then exhausted via exits not shown. The core 
of the cylindrical body is occupied by the various coolant and 
other supplies while the peripheral part is occupied by electrical 
and optical components. A collar 104 permits the body 100 to be 
mounted to a robot in the same manner as the camera and torch 
combination described above. The various fluids, welding wire and 
electrical power and signals are conveyed in the multi-way duct UC 

I as mentioned above. 

The camera, light sources and welding torch thus consolidated 
are readily mounted on a robot for manipulation as required to 
achieve welding. The assembly has a weight of only a few 
kilograms. 

5 The end remote from the connections has a welding gas 

shroud 105 and welding nose 106 of generally conventional form. 
This end is closed by a transparent window disc 107, described in 
. more detail below, and a conical part of the cylindrical body 108 
which tapers to meet the disc 107. The transparent disc 107 forms 
20 a spatter window which resists the action of weld spatter and thus 
protects the inside of the assembly while allowing the infra-red 
illumination from the light source to pass through, in hoth direc- 
tions. The disc 107 is conveniently of a heat resistant glass 
material, e.g. that identified by Registered Trade Mark PYREX, and 
25 designed to last for at least several hours of continuous welding 
operation, being rotatable to give clear areas. The disc is 
easily removable for replacement, for example at the beginning of 
each working shift of say, eight hours. Similarly the shroud 105 
and nose 106 can be replaced as needed. For "tracking" a weld 
30 . path during set-up of the equipment on, say, a master workpiece, a 
spike, not shown, is put in place of the nose and shroud t give 
at its tip the position of welding along the axis of the torch. A 
plurality of. gas ducts, 109, convey the weld shielding gas to 
around the weld. area in known manner. 
35 Inside the cylindrical body 108 are two solid state 

cameras 110, 111 and two light sources one of which, generally 
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indicated at 120, is shown in Figure 11. The cameras and light 
sources are disposed around the periphery of the weld torch core 
at approximately equal intervals. The cameras, as in the embodi- 
ment already described, are aligned parallel to the torch axle at 
a known offset while each of the light sources is inclined in two 
directions to this axis in its respective position. In Figure 10 
axis 130 is inclined at 15° to the torch axis. A further axis 140, 
not shown in Figure 10, lies in an aiming plane including axis 130 
and perpendicular to the plane of the section in Figure 10 and is 
inclined up from this sectional plane at 15°, rising from the 
welding end of the assembly. The two light sources 120, 121 are 
directed in complementary opposite inclinations to the torch axis. 
Other angles between 14° and 18° may be used where appropriate. 

Each light source includes a solid state GaAlAs laser 121 
directed along a respective axis 140 to a cylindrical lens 122. 
The exact position for the laser, to coincide with axis 140, is 
set by precision surfaces formed on a generally circular bulk- 
head 131 in the cylindrical body 108. Guide plates mounted on the 
precision surface permit adjustment of the laser position with 
respect to the lens 122 along axis 140. The cylindrical lens 122 
is mounted in further bulkheads, not shown, to have the axis of 
the lens cylinder in the aiming plane mentioned above and perpendi- 
cular to axis 14. The effect of lens 122 is to produce a sheet of 
infra-red light in a plane including axis 140 and perpendicular to 
the aiming plane which includes axes 130 and 140. This optical 
arrangement produces a sheet of light which diverges through 
window disc 107 and passes the shroud 105 at a compound angle to 
produce a light stripe about 0.3 mm wide visible to a respective 
camera 110 or 111 centrally in the field of view and on a work 
surface at the welding position. Conveniently the shroud may be 
waisted at aid-length to enable the light sheets to "skew" closely 
past the shroud with minimal obstruction. A monitoring device for 
laser 120 can be mounted below the lens 122 in a cut-away part pf 
the lower part of the lens with a view of the laser utput. 

In the illustrated assembly the camera axes are offset 
about 19 mm from the torch assembly axis. The whole assembly 
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produces a compact arrangement of the elements described with 
reference to Figures 1 and 2. The shroud 105 shields the assembly 
from the arc light and weld spatter. The provision of duplicate 
cameras and light sources permits the movement of the assembly 
forwards or backwards without the need to revolve the camera and 
source around the axis and the use of the rearmost camera and 
source to view the finished weld. The disc 107 can be caused to 
revolve when required to move a spattered part from the field of 
view. Two lasers may be used for each light source. 

The electronics circuitry for the solid state cameras is 
mounted on printed circuit cards behind the respective camera and 
on lugs projecting from bulk heads, such as 131, 132, along the 
length of the assembly. The GaAlAs laser diode is a type LCW-10, 
of Laser Diode Lab Inc., injection diode emitting at 830 nm. The 
cylindrical lens is of positive plane cylindrical form, a 
Melles-Grist type 01 LCP 000 reduced to 8 mm. The camera includes 
an optical filter as described above and a Fairchild CCD Area 
Array semiconductor chip type 222, which has 488 x 380 elements. 
Suitable electronics circuitry for extracting pixel information 
from such a chip is known and available and will not be described 
but can with attention to component packing be. produced on printed 
circuit cards small enough to fit inside the assembly. 

The supply of compressed air cools the electronics circuitry 
and keeps out dirt. 

Suitable constructional techniques for the assembly will be 
readily apparent, e.g. light alloy machined castings and fabricated 
parts with a steel cylindrical body. 

Reference has been made above to the limitation in the speed 
of response of the arrangement because of the time taken to process 
the image. The embodiment now described reduces the time to 
process the image so that the response is fast enough for the weld 
to follow the seam at actual welding speeds. 

Referring first to Figure 13 this shows diagrammatically a 
welding area. Two metal parts, PI, P2, are to be welded together 
along a seam line SL by a fillet of arc welding AW applied by an 
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arc AR from a welding head not shown. The welding head is to be 
controlled to follow the seam line SL. The metal parts are 
commonly sheet metal pressings for vehicles, e.g. an axle case. 

The seam line position is detected by shining a stripe of 
light LS to cross the seam line and scanning the area SA with a 
television pick-up device to produce an image by scanning in lines 
along the direction of arrow L. 

Figure 14 shows a typical line of the video signal from the 
scan, in the same sense as arrow L. Immediately after the left 
hand synchronising pulse there is an area of high illumination AL 
from the arc AR. Towards the right hand synchronising pulse there 
-is an area of lower illumination ML produced by metal sputter. In 
between is the signal LSI caused by the light stripe LS. To 
determine the position of the seam and control the welding head it 
is necessary to relate the occurrence of LSL in the video signal 
with the point in time represented by the time based video signal 
and thereby the position on the parts PI, P2. 

Because of the other illumination picked up the position of 
signal LSL is not always well defined. The video information can 
be produced as discrete pixels along each line, typically 256 
Pixels on each of 256 lines and the aim is to relate the occur- 
rence of LSL as closely as possible to a specific pixel in each 
line. Because of the shape of parts PI, P2 the position of LSL 
will vary from line to line. 

Referring to Figure 12 this shows in block schematic form a 
circuit to filter time-based video information to isolate a parti- 
cular component of the information and correlate the occurrence of 
this component with the video time base. 

In the Figure video information is supplied at input VI. The 
video information is conveniently in the form of 256 lines each 
of 256 pixels. The video information is also applied to a frame 
buffer store via connection FB. 

The video information is supplied serially pixel-by-pixei and 
line-by-line to filter F in real time. 

Filter F includes a delay-line DL. Delay— line DL in this 
embodiment is a a 32-tap delay line of the bucket brigade type. 
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Specifically the delay line described in Application Note 105 by 
E.G. & G. Reticon Ltd., California, USA, "A tapped analog delay for 
sampled data signal processing" is used. The various taps are 
indicated generally as DLT each tap having a selected coefficient 
indicated as o.^ — ol ± — <* n , where jo is the total number of 

taps. The outputs from the taps are summed together in element SU 
to produce an output indicating the occurrence of the required 
component . 

This output is applied to a sample-and-hold circuit SAH. The 
operation of the delay-line-based filter F and the sample-and-hold 
circuit SAH are controlled by a clock signal CK. In the present 
embodiment this clock signal is at 5.5 MHz. For each clock pulse 
an amplitude value for the particular component of the video 
signal is provided at the output of the sample— and-hold circuit 
and this appears as an 8-level signal LSS at the output of a 
counter latch LA driven by counter CT. The position of the 
stripe LS, ±n terms of signal LSL in a video line, is thus 
produced in turn for each video line as time-based signal LSS. 

From the practical point of view the clock signal can break 
through the filter as narrow spikes at twice clock frequency. 
These can be removed with a straightforward second or third order 
low pass filter on the output of summing device SU. The bandwidth 
of the output at SU has already been otherwise restricted to 
prevent the low pass filter causing any other problem. Also the 
load impedance applied to each tap must be low enough to reduce 
crosstalk between taps. For a tap output Impedance of about 3 Kohm 
a load impedance of 1 Kohm or less is preferred. Final adjustment 
can be made by adjusting the tap coefficients. The clock rate is 
such that there is negligible signal degradation in the delay 

line. . 

The effect of the filter provided by the tapped delay line 
is indicated in Figures 15 and 16. Figure 15 sh ws a typical 
impulse response for the filter. The filter is of the "DOG" type, 
i.e. difference-of-Gaussians, and can be produced by either soft- 
ware or hardware techniques. However the software technique is 
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not fast enough to give an accurate answer in real time, at leaet 
without restricting the performance of the filter and thereby 
giving rise to an unsatisfactory spatial frequency response and 
degraded signal-to-noise rato. 

An important feature of the filter response used is that it 
has only one large positive peak of impulse response. If a con- 
ventional very narrow-band filter is used to provide the required 
discretion there will be several peaks in the response. This 
could lead to more than one position along a video line being 
indicated for the stripe signal LSL vith obvious problems. Another 
feature is that the time integral of the filter can be arranged to 
be zero and thus there is no response to slow changes in the 
signal. 

The transversal filter having a finite impulse response, of 
which the illustrated filter is an example, is especially suitable 
as its impulse response is readily adjusted. The impulse response 
corresponds with the coefficients of the tap Sl to « n of the delay 
line DL. The mapping from spatial frequency (in ^horizontal 
direction on the work piece) to temporal frequency (in the elec- 
trical video signal) is accomplished by the line scan of the 
pick-up device. 

The arrangement described will thus process a video signal of 
a line-by-line scan across area SA to determine for each line the 
position along the line of stripe LS and provide this position as 
a pixel number along the line without any ambiguity such as two 
pixel numbers spaced apart on the line. 

The arrangement has considerable resistance to interference 
despite the difficult nature of the signal to be processed. 
Considering the 256 pixel line again the stripe LS produced a 
signal LSL which can cover about 6 pixels of the line. (The 
brightness profile of the stripe LS on the workpiece is about 300 
micrometres wide at half-height of its approximately Gaussian 
distribution.) Noise will prevent very narrow-band filters being 
effective because of the wide spatial-frequency bandwidth. Noise 
does appear to be present in most of the spectrum of the stripe 
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signal. However the lower, especially very low, spatial frequen- 
cies can contain a great deal of unwanted information so these 

must be filtered out. 

Figure 16 shows the action of the exemplary arrangement on a 
typical video line waveform. The upper curve is the line waveform 
before filtering, the lower curve that after filtering with the 
stripe signal distinguised from the other signals and background 

noise. 

The arrangement described thus provides in real time 
information about the position of the stripe LS in area SH in 
terms of the pixel number of the signal LSL for each line of the 
scan over area SA. The provision of this information in real 
time, called a "hit-list", permits the control-loop for an auto- 
matically controlled seam tracking welding head to be closed 
especially when welding difficult parts such as sheet metal 
pressings which are likely to vary in their exact shape and fit. 
The gain in the speed of response by using the filter is not lost 
by greater complexity in subsequent calculations on the stripe 

geometry as these do not occupy a large amount of computation 

time. 

Although the embodiments have been described in relation to 
welding the techniques can be applied to other seam or edge following 
procedures for example to control the deposit of glue or sealant 
in relation to a particular feature of an article even when the 
form of the feature varies from a specified shape, say because of 
practical manufacturing tolerances. 
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CLAIMS 

1. Apparatus to control the automatic placing of material along 

a junction between surfaces with reference to the form and position 
of the junction including means controllably movable to deposit 
material progressively along the junction in response to a control 
signal, means linked to the movement of the means to deposit 
material to produce an image of the surfaces and junction, means 
to extract from the image that portion unambiguously defining the 
junction form and position, means responsive to said image portion 
to derive said control signal to control the means to deposit 
material to move in a required manner along said junction to 
deposit material. 

2. Apparatus according to Claim 1 in which the image is an array 
of pixel elements identified by a set of row indices and a set of 
column indices and the extracted portion is an array of pixels, 
one pixel for each of one of said sets of indices. 

3. Apparatus according to Claim 2 in which the image is further 
simplified to at least one straight-line segment defined by end- 
points in the extracted array. 

4. Apparatus according to Claim 1 in which the image is 
processed by a filter to remove from the image information 
produced by illumination at other than a specific wavelength, 
which filter includes a tapped delay to provide a specific impulse 
response. 

5. Apparatus according to Claim 1 in which the material to be 
deposited is a gas shielded arc weld bead. 

6. Apparatus according to Claim 1 in which the means to produce 
an image includes a light source to produce a sheet of light of a 
specific wavelength and a camera to form an image of a region of 
the surfaces and junction including the line of incidence of the 
sheet of light, the sheet of light being inclined to the axis of 
the camera . 

7. Apparatus according to Claim 6 including a welding torch and 
in which the axis of the camera and the axis f the welding t rch 
are parallel and at a specific spacing. 
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8. Apparatus according to Claim 7 in which the axis of the 
source forming the sheet of light is inclined at angles of 15° 
to 18° to the axis of the camera in the plane containing the 
parallel torch and camera axes and in a plane at right angles 
thereto . 

9. A method of controlling an automatic arc welding process at a 
junction of surfaces including producing an image of a welding 
area containing the junction and illuminated by a light sheet, 
extracting from the image that portion representing the form and 
position of the junction and deriving from said image portion 
control information to, control the welding process to operate 
along said junction. 
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